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SECTION  i 


INTRODUCTION 

The  objective  of  the  Ada  Compiler  Validation  Capability  (ACVC)  is  to 
determine  to  what  extent  an  Ada  compiler  conforms  to  the  Ada  Standard.  The 
purpose  of  this  Report  is  to  describe  planned  validation  capabilities,  their 
relation  to  other  stnr.dardardization  activities,  and  our  planned  development 
a  poroac h . 


and 


The  ACVC  is  being  developed  under  a  three  phase  contract, 
3  being  performed  at  the  option  of  the  Government. 


with  phases  2 
The  periods  of 


performance  and  principal  objectives  of  each  Phase  are: 

.  Phase  1  (25  September  '979  through  10  July  '9&Q)  —  to  produce  a 
baseline  set  of  validation  tests,  tools  needed  to  support 
validation  activities,  and  procedures  for  using  the  tests  and  tools 


Phase  2  • 1 1  July  1 98 0  through  10  July  1981  )  —  to  increase  test  set 
coverage  while  maintaining  the  baseline  capability 


Phase  3  C11  July  1981  through  ’0  January  1 932  )  —  to  develop  new 
validation  approaches  that  attack  difficult  validation  problems  not 
fully  solved  in  the  earlier  Phases.  In  addition,  validation 
capabilities  provided  in  earlier  Phases  will  be  updated  and 
maintained  . 


In  the  next  Section 
supported  by  the  ACVC 
provided.  in  Section  3, 


of  this  report,  '  we  describe  the  functions  to  be 
and  the  general  nature  of  the  capabilities  to  b^ 
we  describe  specific  plans  for  each  of  the  Phases. 


In  Section  ^ ,  we  specify  risk  areas.  In  Appendix  A,  we  specify  documentation 


■.a  nr!  a  r  j  a 


a  r  r  1 4  pc  t  c  C  hp  AC  VC . 


SECTION  2 


SYSTEM  SUMMARY 

In  this  Section,  we  describe  the  general  nature  of  the  capabilities 
needed  for  Ada  compiler  validation. 

2 . 2  •  ACVC  Cb  ~  entires  and.  Limitations 

The  Aia  Compiler  Validation  Capability  will  be  used  to  check  how  well  Ada 
language  translators  adhere  to  the  Ada  Report  (the  official  document 
specifying:  the  Ada  Language  Standard).  It  should  be  noted,  however,  that  such 
validation  tests  do  not  form  a  complete  acceptance  test  for  an  Ada  translator. 
Overall  acceptability  encompasses  more  than  compliance  with  a  language 
standard .  Among  the  issues  not  addressed  by  the  tests  that  are  to  comprise 
the  ACVC  are: 

.  the  compiler's  maintainability,  as  reflected  by  its 
.  internal  design  structure 

.  the  implementation  standards  followed  by  its  source  code 
the  quality  of  its  design  and  implementation  documentation 

.  its  interface  with  other  support  tools,  e.g.,  as  reflected  by 
its  object  file  format 

.  its  symbol  table  format  and  options  for  producing  a  binary 
symbol  table  for  use  by  a  symbolic  debugger 
the  internal  format  of  the  program  library 

the  range  of  options  provided  to  compiler  users 

.  its  compile-time  efficiency 

the  helpfulness  of  its  error  message  format  and  content. 

the  efficiency  of  the  object  code  it  generates 

But  two  important  issues  the  ACVC  will  address  are: 

the  effectiveness  of  the  validation  tests,  and 

the  amount  of  effort  needed  to  validate  a  compiler. 


’OhY-1 


I 


L«  B 


The  quality  of  a  compiler  validation  capability  is  ultimately  measured  by 
the  number  and  importance  of  ways  a  validated  compiler  is  later  discovered  to 
be  nonconforming.  if  the  baseline  A CVC  is  of  poor  quality,  Ada  compilers 
containing  significant  errors  may  be  approved  for  official  use.  Once  such 
compilers  are  in  use,  it  is  often  costly  to  make  them  standard-conforming. 
The  r.et  result  is  the  development  of  implementation-defined  Ada  dialects,  and 
the  goal  of  ?.  ToT  Termor  language  will  not  have  been  achieved. 

Consequently,  our  approach  will  apply  several  important  design  principles 
underlying  the  construction  of  high  quality  tests.  Perhaps  the  most  important 
is  that  tests  must  be  designed  so  their  successful  execution  implies  (or  at 
least  makes  it  highly  lixely)  that  certain  implementation  errors  are  absent. 
The  basis  of  this  principle,  which  has  been  argued  at  length  in  [ 1 ]  and  [2], 
is  that  unless  tests  have  been  designed  so  their  successful  execution  rules 
cut  certain  errors,  little  is  learned  from  such  success.  And  of  course,  a 
validated  compiler  is  one  that  has  passed  all  its  tests!  This  means  that  to 
construct  ar.  effective  compiler  validation  capability,  one  must  first 
hypothesize  the  kinds  of  errors  impiementers  may  make  and  then  construct  tests 
that  nan  only  succeed  if  the  errors  are  absent. 

be*  tests  deveiopec  in  advance  of  a  compiler  implementation,  i.e.,  tests 
ie-.-e'.  ,-p*d  wi “ hout  knowledge  of  a  compiler's  internal  structure  (so-called 
"  1 1  a  ■■  k  -  ex"  *c'.stsl  can  no*  ,  in  pr  ire:  pie,  detect  al  1  errors  a  compiler  may 

’  .  Joed'  rough .  •!.  ar.b  ierrart,  L.  ,  "Toward  a  Theory  of  Test  Data 

Delect  ion,"  IEEE-T5E-3E  J_,  2  (June  ’975) ,  156-173- 

i c  [  Dood enough .  J.  r . .  "A  survey  of  Program  Testing  Issues,"  in  Research 
l.ljl5qd_t_iijn.?.  -J!  wa_re.  Teer.rpluy.v ,  P.  Wegner,  «d . ,  MIT  Press,  Cambridge, 
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contain,  even  though,  strictly  speaking,  any  errors  constitute  non-conforming 
behavior.*  Since  not  all  errors  can  be  detected,  our  objective  in  designing 
black  box  tests  will  be  to  ensure  that  the  cost  of  developing  and  executing 
tests  intended  to  detect  certain  errors  is  not  greater  than  the  benefit  of 
detecting  the  error.  For  example,  undetected  errors  are  non-critical  if  they 
are  unique  to  a  particular  application  program  or  programmer  and  an 
alternative  use  of  the  language  provides  a  satisfactory  "write-around"  until 
the  erro1"  is  fixed.  Such  errors  usually  reflect  the  use  of  rare  combinations 
of  language  features.  Moreover,  although  many  implementation  errors  arise 
from  failure  to  deal  correctly  with  certain  combinations  of  language  features, 
we  will  not  attempt  to  detect  errors  judged  unlikely  to  occur  in  practice  or 
whose  effect  is  unlikely  to  be  critical.  To  achieve  this  objective,  the  tests 
we  develop  will  be  based  on  a  thorough  analysis  of  wide-ranging  implementation 
problems  and  important  interactions  among  language  features  that  must  be 
successfully  dealt  with  by  every  conforming  implementation.  Tests  will  then 
be  devised  to  check  whether  these  difficulties  have  been  dealt  with  properly. 

Since  the  validation  tests  are  to  be  used  for  compilers  executing  in  a 
variety  of  host  environments  and  generating  code  for  a  variety  of  target 
environments,  the  tests  must  be  easily  adapted  to  cope  with  these  differences. 
This  means  minimizing  the  amount  of  manual  effort  required  to  adapt  tests  to 
host  and  target  environments  and  to  analyze  the  results  of  test  compilations. 


*  The  possibility  of  developing  tests  based 
internal  structure  will  not  be  addressed  at  1 
effort,  ar.d  possibly  not  until  Phase  3 ,  since 
■'I  i  t ' f'  j  ■-'ul  \  'iTid  eo^M  v  *\  ■>  —on? t  rue  t. 


on  knowledge 
east  until  Phase 
such  test  s  a  j 


of  a  compiler's 
2  of  our  planned 
currently  very 
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2.2.  ACVC  Struc tur e 


The  ACVC  ha?  three  main  components : 

.  An  Implement er 1 s  Guide  describing  explicitly  the  implications  of 
the  Ada  Report.  It  will  specifically  address  implementation  issues 
that  might  be  overlooked  by  an  unwary  implementor .  The  purpose  of 
this  Guide  is  partly  to  point  out  the  implications  of  statements 
made  in  the  Ada  Report  and  partly  to  specify  conditions  to  be 
checked  by  validation  tests. 

Tes t  programs  to  be  submitted  to  a  compiler. 

.  Validation  support  tools  that  assist  in  preparing  test  s  'hr 
execution  and  in  analyzing  the  results  of  executions. 

These  components  are  described  in  more  detail  below. 


2 . 2 . 1 .  The  Implementer 1 s  Guide 


The  primary  document  to  be  produced  in  Phase  '  is  the  Implementer ’ s 
Guide.  This  document  describes  the  ramifications  of  the  Ada  Report  and  in 
particular,  describes  implementation  issues  that  must  be  properly  addressed  in 
a  production  compiler  design.  The  document  also  specifies  objectives, 
constraints,  and  guidelines  for  developing  specific  validation  tests.  The 
document  will  be  Keyed  to  the  subsections  of  the  Ada  Report.  For  each 
subsection,  there  will  be  an  analysis  treating  the  following  topics: 


Ramifications  of  the  Semantics  --  This  section  will  document 
ir.pl  icat ; ons  of  the  language  semantics.  In  cases  where  the 
implied  semantics  are  derived  from  statements  in  separate  sections 
of  r  r,‘-  Ad  i  Report ,  a  rationale  for  the  interpretation  will  be 
*iver>  t--  : ‘cose  who  are  not  intimately  familiar  with  the  Report 
or  the  rationale  underlying  the  Ada  design.  ke  will  g  ive 
part  t  *>>;  -  -on  i:  ’  refractions  between  Ada  constructs..  For 

ex  amp  L«  .  for’  -•■y.  faction  b  .  1 .  .  Array  and  slice  Assignments)  ,  we 

wou- d  point  out  tro.t  ■  f  A  and  R  are  strings  with  index  ranges 


t 


are  all  legal  assignments.  The  first  assignment  does  net  raise 
the  OVER LA ?_ER F.C’R  exception  since  the  result  of  concatenation  is 
r.ct  a  slice  value,  and  hence  there  are  no  index  rang,^s  that 
overlap.  T: -  second  assignment  is  legal  because  the  number  of 
element  s  1 1 ,  -.•••re.  is  t he  same  and  there  is  no  requirement  that 
the  bounds  of  an  empty  slice  satisfy  the  range  constraints  for 
indexing  elements  of  the  array.  The  third  example  shows  a  'ase 

where  DYER  DA r _ rlh r DR  should  not  be  raised  even  though  the  slices 

have  a  value  in  common. 


Dcmrlle-t:  r.e  Constraints  —  This  section  will  explicitly  state 
cent  '■ x  i  t  ive  syntactic  and  semantics  constraints  to  he 

:  r  -'-a  ■  c  c  .  m.  pale- time  or  lead-time),  i.e.,  it  will  detail,  t  he 

:.m:  1  lest  ;_en?  of  the  legality  constraints  stated  broadly  in  the  Ada 
Ft  t-  "Z .  listing  these  constraints  explicitly  provides  a 
cor.veni ant  che.'Kiist  for  implementors  to  ensure  they  cover  all  the 
cortex*:- scr.si*  ive  validity  constraints  implied  py  tr.e 
specification  and  also,  of  course ,  suggests  validation  tests  to  be 
provided.  When  useful,  we  will  note  non- straight  forward  ways  the 
constraints  car  be  violated. 


Exception  Conditions  --  This  section  will  explicitly  state  what 
exception  conditions  may  be  raised,  since  these  determine  what 
run-time  checks  implement ers  must  provided.  For  example,  in  the 
case  of  beet  lor  5 . 1 . 1  .  we  would  note  that  RANGE_ERROR  and 
ACCESSJERHOF  v  when  assigning  to  a  record  component)  are  the  only- 
exception  cond it  ions  explicitly  associated  with  Section  5.1.0. 

Test  Cb.i ectives  and  Design  Guidelines  --  This  section  will  state 
test  objectives,  give  cesign  guidelines  for  test  case 
construction,  list  implementation  errors  to  be  checked  for,  and 
list  problems  to  be  kept  in  mind  while  designing  cr  writing  test 

cases  (e.g.,  to  ensure  that  the  tests  are  as  portable  as  possible 

and  that  they  are  effective  in  determining  the  actual  pehavior  of 
a  compiler'.  Borderline  casp-s  to  be  considered  by  ?r. 

implementation  will  be  noted  here  with  program  examples  when  n.  -  r 

examples  are  the  most  straightforward  way  of  illustrating  *  •  - 
point  being  made.  These  examples  are  candidates  for  inc.2  osier  :  * 
"he  valid -at  lor  test  set. 


D_n_p_s  in.  T_hf?  An  a  1  vs  1  s  --  This  section  will  document  those  a  r  i.  >  :■  t - 
of  the  Ada  specification  for  which  test  objectives  nave  not  ye* 
been  coflr.e  ■  ,  r  dd0"  because  cf  potential  changes  in  t n<-  language 
•'esi'/r  or  because  it  is  not  c  lea**  how  to  con  struct  tests  that.  will 

the:;1'  as  pc  :*s  tr.e  AC  VC  for  which  further  work  is  reeded  ,  e  it  her 
later  ^'nas?  ! .  or,  if  the-  problem  is  •  research  problem ,  :  t 

explains  the  issues  for  which  further  study  is  required.  In 
ad  di  tl.cn .  ‘.'.is  section  will  list,  questions  for  which  tr.e 

57*  ' .  r  ■  '(;r.c  not  orev  j.  <1  ob-v  ?  ouf  inpwrrr  .  ■.ur-'  ini*  ;  n  1 
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2.2.2.  Val  idatior.  Test.  Structure 

One  issue  ir.  develop  in.'  a  so;,  of  va'.i.1at  icn  tests  is  whether  t -•  -  5 

should  be  few  in  numter,  which  irr.pl  ie?  a  given  test  can  fail  for  many  reasons , 
or  whether  individual  tests  should  focus  on  a  small  set  of  implement  at  lor; 
erne  r  ?  ,  t  p.p  y  '  r-.d.'  i  t  o  i  r  number  of  tost .°.  to  y  r  o  ^  Ci  ?.  s d  by  s 


m 

* 


avinv  ••.  large  number  of  fairly  special  ir^-d  tests  is 
is  obtainec  about  compiler  flaws.  Although  validations 
AIL  ...  ,e.,  either  the  compiler  conforms  to  the  language 
’  t )  .  ir.  practice,  such  a  response  tc  2  validation  is  as 
.er  that  terminates  compilation  with  the  single  message 
fust  as  compilers  are  designed  to  detect  more  than  one 
it  useful  for  validation  tests  to  reveal  as  many 
of  a  compiler  a3  possible.  There  are  several  reasons 
cm  oiler  being  validated  may  be  needed  for  immediate  use 
be  cot,  r»i  L  er 1  s  non  con  t  enmities  3  re  miner  with  resreet  to 


;  *  'jcn  Lo!  t o y  Qpr»'.Je  it  i ?  ooe  t— ol'tect.  ivo  tc  permit  use  of  the 

.  r, bcT si  1  e*p  wh i !. **  the  errors  are  beirp1  corrected  .  The  cue ion  o f 


emitted  ir*  r-o -icy  decision  b-i sod  or'  t he  •is* 
Lor  arid  *  be  *-x  rioted  ore  of  th»'  ^ompi  L  ^r .  due 
W  ga  r*r*  p  —  C>r"  —  *  f  .  {  hv  ’**  '  "I  *  IT  7  "  r  ^  *  ’  •'  1  *  *  '  ’*  r1'*',  ;  ► 
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SSagOS, 

'pr  r f.  ■*  *"  - 


no u g r.  s z r  i e  t  „y  s p e akine  ,  s r.  -  o  a 
chair. ted  program  does  not  -con  for" 
rt  in  ?  i I ent  regarding  what  further 


, t s .  These  tests  also  demonstrate  the  rejection  of 
trograns,  bet  the  errors  in  these  tests  may  be  defected 
or  load  time  by  some  imrlerr.entatior.s .  These  tests  are 
e‘  he--  because  their  PATe/FAIL  criterion  is  different 
•r  .'.ass  T:  tests,  namely  the  test  succeeds  if  an  error 
d  oo fc r»  execution  begins.  An  example  of  a  class  BL 
-  temp:.  .  irm  a  version  of  a  package  with  revised 
enter facer  foil  owed  oy  an  attempt  to  load  the  package 


'di Tying  the  ur.  i  t  s  using  tne 


ease . 


hose  tests  are  designed  to  be  execute t  after 


v-‘-r;fv  tna 


.  Off*  S  J 


p  o *y_  m  'J  p  p  ex  P  0  u  e  C 


,  q  r  p  :v  t>  e r  1  f  1  e  .  j  sernsn  f  ic  s .  7  he  p  r  oz>  r  an)  s  are 

insofar  as  possible'  arc  generate  PAT FAIL  messages 


P»'  r;  r-  r» 

>'"f  1  ~  PY*-~  ' 


are  used  to  check  capacity  reaui renor4;  ? 
at  toe  cumber  of  subscript?,  ca r c;r. i: v ° ** ? 

r : IT: b o r  o f  1  irk- ec  it  s yrr. t- o  1  r  ,  o  t . '  .  : r 

5*  *•  #•' "■  p  a *c ,t; p ~  y  -■  £■  p  pa  b  r 0  r  t  r  c  r  .i  r  y  ■  4  ^  .  p  -  ^  y 


result  of  generic  instantiation  or  inline  substitution  of 
subprograms.  Other  aspects  of  an  implementation  that  affect  its 
interface  with  the  external  object  computer  environment  but  which 
are  not  currently  specified  are  also  checked  by  these  tests,  e.g., 
consider 


type  BIASED  is  new  INTEGER  range  0 . . 1 5 ; 
for  BIASED  use  4; 

In  interfacing  with  external  devices,  it  could  be  important  to  know 
whether  2#1000  represents  the  value  zero  (if  a  signed  biased 
representation  is  used)  or  the  value  8  (if  an  unsigned 
representation  is  used) .  Other  tests  in  this  class  will  be  used  to 
determine  what  interpretation  an  implementation  has  taken  with 
"espect  to  ur. intended  ambiguities  discovered  in  the  Ada  Report  . 
The  results  cf  these  tests  are  used  to  inform  compiler  users  and  to 
improve  the  Report. 

•  Class  £  Tests .  These  are  tests  that  demonstrate  the  correct  and 
complete  operation  of  standard  packages,  e.g.,  the  INPUT_OUTPUT 
package,  the  mathematical  library  package  (when  one  is  specified  as 
a  standard),  etc. 

Additional  test  Classes  may  be  added  as  a  result  of  Phase  2  and  Phase  3 
efforts,  e.g.,  tests  of  object  code  efficiency,  compiler  performance,  etc. 


Within  each  class,  a  test  will  be  given  a  name  corresponding  to  the 
Chapter,  Section,  and  Subsection  number  of  the  part  of  the  Report  to  which  the 
test  pertains.  Since  most  subsections  require  more  than  one  test,  tests  will 
be  assigned  sequential  number?  within  subsections.  Each  source  code  module 
will  be  in  a  separate  file  which  will  be  given  the  name  of  the  test.  There 
may  also  be  a  number  of  INCLUDE  files  used  in  conjunction  with  test  modules. 

2.2.3.  Validation  Suppo rt  Tools 

There  are  conceptually  three  environments  relevant  to  validating  a 
compiler : 

the  validation  host  (VH )  environment,  in  which  validation  tests  are 
prepared  for  execution  by  an  Ada  compiler  and  validation  results 
are  analyzed: 
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the  compiler  host  (CH)  environment,  in  which  Ada  tests  are 
compiled.  In  some  implementations,  target  modules  may  also  be 
loaded  in  the  CH  environment;  executable  code  is  then  transported 
to  the  object  computer  for  execution; 

the  ob ject  host  l OH)  environment  (the  target  computer)  in  which  Ada 
programs  are  executed  . 

An  important  aspect  of  our  approach  to  the  ACVC  is  the  Validator  Host  concept. 
The  Ada  validation  tests  will  constitute  a  fairly  large  amount  of  textual 
material  .  Powerful  text  editing  and  file  management  tools  are  needed  to 
create,  access,  and  maintain  this  material.  More  importantly,  when  preparing 
to  perform  a  compiler  validation,  the  tests  must  be  adapted  to  conform  to  the 
conventions  of  the  Compiler  Host  for  submitting  text  to  the  compiler  and 
obtaining  output.  In  addition,  a  variety  of  test  parameters  may  be 
Implementation  dependent  (e.g.,  the  maximum  and  minimum  integer  literal 
values).  Appropriate  values  must  be  inserted  into  some  tests. 

All  these  text  handling  problems  will  be  dealt  with  by  the  validator  host 
tools,  which  will  consist  primarily  of  text  editing  tools  and  macros.  If  a 
SNOBOL  processor  is  not  available  in  the  validator  host  environment,  we  will 
used  TECO  and  TECO  macros  to  perform  commonly  needed  text  manipulations.  If 
the  compiler  host  is  available  on  the  ARPANET,  the  validation  tests,  once 
adapted  to  the  CH  requirements,  will  be  sent  to  the  CH  using  the  Net.  If  the 
CH  is  not  on  the  Net,  the  tests  will  be  brought  to  the  host  using  an 
appropriate  physical  medium.  If  a  comparable  ARPANET  host  is  available,  it 
might  be  used  to  transfer  the  text  to  such  a  medium. 

Text  editing  tools  will  also  be  used  to  analyze  the  machine  processable 
results  of  compilations  and  executions  --  to  confirm  that  all  required  tests 
were  submitted  and  that  all  test  failures  are  identified. 
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The  text  processing  tools  we  provide  will  be  documented  in  accordance 


with  the  specifications  provided  in  Appendix  A. 

2.2.4.  Val idation  Procedures 

Usage  of  the  tests  and  tools  will  be  described  in  a  Validation  Procedures 
Manual  structured  in  accordance  with  the  specification  cited  Appendix  A.  The 
general  nature  of  these  procedures  is  discussed  in  Section  2.3* 

2.2.  Validation  Activities  to  be  Supported 

There  are  two  principal  activities  supported  by  the  Ada  Compiler 
Validation  Capability:  1 )  compiler  validation  and  2)  revising  the  validation 
tests,  tools,  and  procedures.  In  this  Section,  we  describe  the  general  nature 
of  these  activities  since  they  affect  the  structure  and  content  of  the  ACVC. 


Val idatine 


Compiler 


There  are  five  principal  activities  performed  in  validating  a  compiler. 
These  are: 


1 .  Collecting  information  needed  to  select  and  adapt  the  validation 
tests  for  the  compiler  being  tested. 

2.  Preparing  the  tests  for  submission  to  the  compiler. 

3.  Performing  the  tests,  i.e.,  compiling  them,  executing  the 
executable  tests,  and  collecting  the  results  for  later  analysis. 

4.  Summarizing  the  test  results. 

5.  Archiving  the  findings. 


2 . 3 . 1 ■ 1 .  Collecting  Information 


H 


The  principal  output  of  this  activity  is  a  Test  Plan  describing  all  the 
information  needed  to  prepare  and  conduct  the  validation.  Information  is 
needed  regarding: 

the  language  accepted  by  the  compiler,  e.g.,  the  character  set 
accepted  by  the  compiler  (full  ASCII,  EBCDIC,  63-character  CDC, 
etc.),  allowed  length  of  input  lines,  how  an  "end-of-line"  is 
defined,  the  semantics  and  restrictions  on  representation 
specifications  (Section  13  of  the  Preliminary  Ada  Reference 
Manual),  the  syntax  and  semantics  of  commands  for  manipulating  the 
program  library  (Section  10.iO,  any  language  subsetting,  the  number 
of  levels  of  integer,  fixed,  and  floating  point  precision  supported 
and  their  names,  ranges  associated  with  numeric  types,  etc.  This 
information  is  used  in  selecting  and  adapting  the  tests  to  be  used, 
e.g.,  some  tests  do  not  apply  if  only  a  single  integer  precision  is 
supported . 

compiler  options  provided,  which  ones  are  to  be  tested,  and  how 
they  are  exercised. 

.  the  compiler  host  environment,  including  file  structuring 

capabilities,  file  naming  conventions,  the  method  of  invoking  the 
compiler,  the  method  of  transferring  object  code  for  execution  on 

the  target  computer,  the  methods  for  linking  and  loading  modules  to 

be  executed ,  the  method  for  collecting  and  displaying  compilation 
results ,  etc  . 

.  the  object  host  environment,  including  its  file  structuring 

capabilities,  file  naming  conventions,  the  method  for  linking  and 
loading  (if  this  is  to  be  done  on  the  target  computer),  the  method 
of  initiating  object  code  execution,  the  method  for  collecting 
execution  results  in  machine-processable  form,  etc. 

In  addition,  if  the  validation  is  being  performed  on  a  new  version  of  a 

previously  validated  compiler,  information  from  the  previous  validation  must 

be  analyzed  to  determine  how  revalidation  costs  might  be  minimized  (e.g.,  by 

combining  test  program?  into  larger  modules,  by  reusing  test  preparation 


macros  ,  etc  .  ;  . 
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This  information  is  used  to  prepare  a  comprehensive  Test  Plan  describing 
what  tests  are  to  be  run,  information  needed  to  prepare  tests  for  submission 
to  the  compiler,  estimates  of  time  required  to  perform  the  tests,  procedures 
for  manually  and  automatically  analyzing  test  results,  etc.  (see  Appendix  A). 
The  Test  Plan  controls  subsequent  phases  of  testing  activities.  The  ACVC  will 
include  a  orobotype  Test  Plan  to  serve  as  a  guide  for  creating  actual  Test 
Plans . 


2 .  * .  1  . 2  .  Preparing  Test  s  for  Submission 

The  basic  purpose  of  this  activity  is  to  adapt,  and  in  some  caes, 
generate,  text  files  ir  a  form  suitable  for  submission  to  the  compiler  being 
validated.  A  number  of  ACVC  text  editing  macros  (and  macro  templates)  will  be 
provided  to  automate  as  much  of  the  preparation  process  as  possible. 

In  general  ,  the  following  steps  must  be  performed  before  the  Ada 
Validator  test  files  are  transported  to  the  compiler  host: 

.  generate  specific  versions  of  tests  for  the  full  range  of 

capabilities  supported  by  the  compiler  (e.g.,  to  test  all  levels  of 
numeric  precision  supported,  including  interactions  among  these 
levels ) ; 


substitute  implementation-dependent  parameters  in  appropriate 
tests,  e.g.,  literal  values  testing  the  upper  and  lower  limits  of 
the  numeric  ranges  supported,  string  literals  that  contain  all 
characters  of  the  supported  character  set,  etc.  In  general,  the 
validation  tests  will  contain  the  equivalent  of  text  macro 
invocations  so  implementation-dependent  constructs  can  be 
substituted  at  the  appropriate  places. 

embed  the  tests  in  the  appropriate  job  control  language  for 
compiling,  linking,  and  executing  the  tests. 

generate  file  names  that  satisfy  the  compiler  host's  naming 
conventions . 

tailor  Class  D  tests  to  the  correct  size. 
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prepare  text  processing  macros  suitable  for  analyzing 
machine-processable  results  of  compilation  and  execution. 

A  file  of  implementation-dependent  parameters  must  be  prepared  for  every 
validation.  This  file  gives  values  for  numeric  precisions,  values  of 
attributes  of  built-in  types,  and  other  parameters  to  be  used  in  generating 
tests  that  are  appropriate  for  a  specific  compiler.  An  important  aspect  of 
the  AC VC  test  set  design  is  identifying  places  where  these  parameters  should 
be  usee  ir.  specific  tests.  Unless  these  parameters  are  used  appropriately,  it 
will  be  difficult  to  adapt  the  tests  to  the  capabilities  of  a  specific 
compiler . 

Difficulties  in  using  the  test  preparation  editing  tools  will  be  noted 
and  input  to  the  Tool  Revision  process. 


The  files  prepared  in  the  previous  step  are  transported  to  the  compiler 
host  and  compiled.  All  object  code  except  for  Class  C,  E,  and  F  tests  is 
discarded.  Listings  from  all  compilations  are  saved  in  machine  readable  form 
and  processed  on  the  Validator  host  computer  to  check  that  all  intended  tests 
were  compiled  and  that  expected  results  were  achieved. 


The  object  code  is  executed  on  the  target  computer  and  results  are  saved 
in  machine  readable  form  for  automatic  analysis  on  the  validator  host.  Since 
people  are  notoriously  unreliable  in  scanning  results  for  correctness, 
extensive  effort  will  be  directed  during  test  case  design  to  ensure  results 
can  be  automatically  checked  insofar  as  possible. 


Any  difficulties  encountered  during  compilation  or  execution  due  to 
errors  in  the  tests  themselves  will  be  noted  and  used  as  inputs  to  the  Test 
Revision  process. 

2.^. 1 .4.  Summarizing  Results 

Using  editing  tools  and  report  templates  provided  by  the  ACVC,  results  of 
a  validation  will  be  summarized,  following  the  report  structure  tentatively 
outlined  in  Appendix  A.  The  purpose  of  the  Validation  Summary  Report  is  to 
concisely  describe  what  comDiler  was  validated  and  what  nonconforming 
behaviors  were  discovered.  Since  groups  of  tests  may  fail  for  a  common  cause, 
preparation  of  a  validation  summary  report  is  not  a  completely  automatable 
task.  Results  must  be  studied  to  extract  as  much  useful  information  as 
possible . 

2. S. 1 . 5 .  Archiving  Findings 

As  a  last  step,  all  plans,  tests,  preparation  aids,  post  processing 
analyzers,  and  findings  are  archived  for  later  use  when  a  revised  version  of 
the  compiler  may  need  to  be  revalidated.  The  ACVC  will  recommend  archiving 
procedures  and  concepts. 

2.4,  Revising  the  Val idation  Capabilities 

In  essence,  the  validation  capabilities  consist  of: 
the  validation  tests 
the  Implementer ' s  Guide 
validation  support  tools 

.  descriptions  of  procedures  for  using  the  tools 


There  are  a  variety  of  reasons  for  modifying  any  of  these  capabilities: 

.  problems  encountered  in  conducting  a  validation  may  lead  to 
modifications  in  the  validation  support  tools  and/or  the  test  cases 
(to  remove  test  case  errors). 

.  errors  discovered  in  validated  compilers  may  indicate  previously 
undocumented  implementation  problems;  both  the  Implementer ' s  Guide 
and  the  validation  tests  must  be  updated. 

.  ambiguities  are  discovered  in  the  language  specification. 
Additional  Class  E  tests  are  needed  to  determine  how  a  given 
implementation  has  resolved  these  ambiguities. 

.  the  validation  tests  are  expanded  to  cover  known,  weaknesses. 

.  the  language  specification  is  modified. 

After  the  Baseline  Validation  Capability  is  delivered,  it  will  be  placed  under 
configuration  control.  Procedures  will  be  specified  for  producing  new 
releases  of  the  tests,  tools,  and  associated  documentation  (see  Appendix  A). 


ic 67-* 


SECTION  3 


PLANS  FOR  EACH  PHASE 

In  this  Section,  we  describe  our  current  plans  for  capabilities  to  be 
provided  in  each  Phase  of  the  effort. 

3 .  2  ,  Pha se  Plans 

Tne  purpose  of  Phase  1  is  to  produce  a  baseline  validation  capability 
that  addresses  all  aspects  of  Ada  to  some  extent  and  whose  weaknesses  in 
coverage  are  documented.  A  prime  focus  in  Phase  1  will  be  the  Implementer ' s 
Guide.  A  high  quality  validation  test  set  (i.e,  one  that  detects  even  subtle 
nonconformities)  must  address  interactions  among  language  features  (since 
these  are  a  primary  source  of  implementation  problems  and  subtleties)  and  must 
cover  potential  specification  misinterpretations  and  subtle  ramifications. 
Pointing  out  implementation  problems  explicitly  will  be  of  considerable 
assistance  in  discouraging  Ada  dialects.  Often  dialects  arise  because  of 
inadvertent  misunderstanding  of  a  specification.  If  the  Implementer ' s  Guide 
points  out  possible  mistakes,  implementers  are  less  likely  to  make  them. 
Hence,  producing  a  thorough  implementer ' s  guide  is  necessary  not  only  to  guide 
the  construction  of  high  quality  tests  but  also  to  help  prevent  inadvertent 
deviations  from  the  specification. 

The  approach  we  are  taking  in  Phase  1  is  to  make  three  iterations  in 
producing  the  Implementer ' s  Guide.  The  iterations  are  needed  because  Ada  is 
currently  undergoing  revision,  with  a  draft  revised  specification  due  1  March 
and  a  final  specification  due  1 5  May  ^SO.  To  cope  with  the  possibility 
of  changes,  the  first  iteration  will  address  only  the  most  stable  parts  of 


Ada . 


A  draft  Implementer ' s  Guide  for  the  stable  subset  of  Ada  will  be 


available  in  January  ^80 .  An  intermediate  draft  for  the  second  iteration 
will  be  available  in  March  198O.  A  first  draft  for  the  third  iteration 
(covering  all  of  revised  Ada)  will  be  available  in  May  ^980.  The  final 
version  will  be  delivered  at  the  end  of  Phase  1  in  July  It  is  our 

intent  00  permit  limited  distribution  of  these  intermediate  drafts  to  people 
designated  by  OAPPA  and  to  hold  design  reviews  at  which  recipients  of  the 
drafts  will  be  invited  to  comment.  In  this  way,  we  will  identify  errors  and 
omissions  in  our  analysis  and  the  validation  effort  will  benefit  from  the 
experiences  of  those  engaged  in  experimental  or  production  implementations  of 
Ada  compilers. 

Not  all  areas  of  Ada  can  be  covered  with  equal  thoroughness  in  the 
baseline  tests.  In  particular,  more  tests  are  needed  to  the  extent  an 
implementer  has  many  ways  of  realizing  a  required  capability.  The  greater  the 
variety  of  implementation  approaches  and  requirements,  the  greater  the  number 
of  tests  needed  to  check  that  each  approach  has  been  carried  out  successfully. 
There  are  several  areas  where  Ada  leaves  considerable  implementation 
flexibility.  These  areas  will  not  be  covered  thoroughly  in  Phase  1 .  They 
are : 

tasking  semantics  for  a  wide  range  of  target  computer 

architectures ; 

.  the  INPUT_0UTPUT  package  for  the  full  range  of  I/O  peripheral?  Ada 
compilers  might  support; 

.  the  correctness  of  optimizations 

other  standard  packages  that  may  eventually  be  defined,  e.g.,  the 
mathematical  functions  package. 

Only  basic  capabilities  in  these  areas  will  be  covered  in  Phase  1 . 
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In  Phase  1 ,  we  will  also  produce  a  basic  set  of  tools  for  adapting  tests 
to  the  requirements  of  a  particular  compiler,  and  we  will  provide  a 
description  of  procedures  to  be  followed  in  using  these  tools  to  validate  a 
compiler . 
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Tne  areas  specifically  deferred  for  thorough  coverage  in  Phase  1  will  be 
addressed  in  Phase  2.  Ir.  addition,  the  baseline  coverage  will  be  updated  and 
improved  based  on  further  analysis  of  the  official  Ada  Report.  Since  the 
official  Ada  Report  is  scheduled  for  approval  only  two  months  before  the  end 
of  Phase  1 ,  it  is  likely  that  further  analysis  in  Phase  2  will  show 
deficiencies  in  the  Phase  1  test  coverage  that  were  not  recognized  at  that 
time.  These  deficiencies  will  be  corrected  in  Phase  2,  and  all  associated 
documentation  updated. 

A  Key  Phase  2  activity  will  be  to  use  the  baseline  tests  in  validating  an 
experimental  or  production  compiler.  Past  experience  with  validation  tests 
indicates  that  even  when  considerable  care  has  been  taken  in  test  design,  more 
problems  will  be  discovered  when  the  validation  capability  is  used  the  first 
time  than  when  it  is  used  on  subsequent  occasions.  Correcting  these  problems 
will  be  an  important  Phase  2  activity. 

3 .  3. •  Phase  3. 

Phase  3  will  address  research  issues  to  further  refine  the  validation 
capabilities.  Although  it  is  difficult  at  this  time  to  accurately  predict 
what  these  issues  will  be,  it  is  likely  that  work  will  continue  to  improve  the 
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capability  for  validating  Ada's  tasking  semantics  and  for  ensuring  that 
optimizations  are  performed  correctly. 

In  addition,  it  is  likely  that  the  Ada  standard  will  be  updated  during 
the  Phase  ?  period  to  reflect  the  experience  gained  in  producing  the  first  Ada 
production  compilers  and  in  using  Ada  for  some  embedded  computer  applications, 
"re  validation  tests  will  be  updated  to  reflect  these  changes  as  well  as  any 
-  f '.  ? ier.o ;  iseovered  in  the  course  of  validating  various  Ada  compilers. 
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SECTION  A 
RISK  AREAS 

The  primary  area  of  risk  in  Phase  1  is  that  the  schedule  for  revising  Ada 

will  slip.  If  the  1  March  1 980  date  for  delivering  a  draft  of  the  Ada  Report 

r_  :  r  the  '?  May  *9p0  date  f-r  delivering  the  final  version  of  the  Ada 

-►--s .  rt  slips,  it  will  be  difficult  to  ensure  that  the  planned  baseline 
■  .  ..s  achieved  by  'J  July  1980.  If  the  possibility  of  a  slip  is 

ip*-:’ rifled  sufficiently  early,  it  may  well  be  possible  to  lengthen  Phase  ■  by 
ar.  ecual  amount  without  incurring  substantial  additional  cost. 

A  secondary  area  of  risk  is  that  if  the  Preliminary  Ada  Reference  Manual 

is  found  to  be  very  difficult  to  interpret,  a  large  number  of  questions  will 

have  to  be  asked  of  the  language  designers  and  answers  received  before 
analysis  of  implementation  difficulties  and  test  objectives  can  be  undertaken 
productively.  Cur  initial  analysis  of  some  of  the  seemingly  simpler  areas  of 
the  language  indicates  that  this  may  pose  a  problem.  Our  plan  is  to  identify 
such  questions  as  quickly  as  possible,  so  work  can  proceed  on  those  areas  of 
the  language  where  the  intent  is  clear. 
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Documentation  Formats 


This  Appendix  describes  the  formats  to  be  used  in  documenting  the 
following  validation  capabilities  in  each  Phase  of  the  effort: 

,  the  Implementer ' s  Guide 
validation  support  tools 
test  programs 
validation  procedures 


The  content  of  the  Implementer ' s  Guide  will  follow  the  structure 
indicated  in  Section  2.2. 1  of  this  document.  It  will  be  formally  updated  in 
Phases  2  and  3  of  this  effort. 


The  validation  support  tools  (i.e.,  the  text  editing  macros)  will  be 
documented  in  accordance  with  DoDI  7935. 1-S,  dated  1 3  September  ^  97 7 ,  using 
the  specification  for  a  Program  Maintenance  Manual. 

The  set  of  test  programs  will  also  be  documented  in  accordance  with  DoDI 
7?3=;.1-S,  using  the  specification  for  a  Program  Maintenance  Manual.  However, 
sections  2.4e-h  and  3  do  not  apply.  The  functional  description  of  each  test 
program  (Section  2.^b)  will  reference  the  appropriate  test  objectives  section 
of  the  Implementer ' s  Guide. 


Procedures  for  using  the  validation  support  tools  and  tests  to  validate  a 
compiler  will  be  described  using  DI-M-B^O,  Users  Manual  (Computer  Program). 
An  part  of  these  procedures,  a  sample  template  for  a  Test  Plan  will  hr 
pr  v  vied  ,  fol  lowing  the  general  outline  of  D1 -T-370?A .  CPC!  i'en' 


Plans/Procedures.  The  format  of  the  Validation  Summary  Report  will  be  adapted 
from  the  format  currently  used  for  COBOL  Validation  Summary  Reports  provided 
by  the  Federal  COBOL  Compiler  Testing  Service. 


END 
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